Today we have news of a confirmed security vulnerability that impacts tens of millions of legacy Dell EMC servers. This vulnerability is known and is a broader industry-wide issue that impacts far more than Dell EMC iDRAC. We are calling this one iDRACula short for “integrated Dell Remote Access Controller unauthorized load access.” Earlier this week, a user on the STH forums posted that along with another individual, they were able to bypass the Dell EMC iDRAC firmware protections and load their own custom firmware onto the iDRAC baseboard management controller both via local access and via a remote access method. If exploited by a malicious party, iDRACula can allow a malicious actor to have complete remote administrative capability of a server.
In this article, we are going to talk a bit about the iDRACula vulnerability, and why it is a big deal. We are going to discuss the process we went through this week that led to this article. We are going to interview the STH forum member who was able to successfully use the vulnerability to load custom firmware on the Dell EMC iDRAC. Dell EMC quickly verified what has happened, and has put together mitigation steps along with providing a statement to STH for this article. Finally, we are going to discuss why this is a broader industry concern.
The TL;DR of iDRACula
Virtually every server today is shipped with an embedded computer that allows for remote administration. These embedded computers are referred to as baseboard management controllers or BMCs and are found in devices ranging from servers to networking devices and even server storage shelves. If you want to learn more about BMC’s and their functionality, we have a short piece: Explaining the Baseboard Management Controller or BMC in Servers.
Dell EMC PowerEdge servers are one of, if not the, most respected brands in the server industry. PowerEdge management uses their BMC and a software management solution called iDRAC (integrated Dell Remote Access Controller.) The iDRACula vulnerability involves replacing the signed iDRAC firmware with an altered firmware package, circumventing numerous protections that the Dell EMC team has in place for its older generations of servers. With advances in BMC hardware and Dell EMC’s security efforts, iDRACula is not a vulnerability on 14th Generation PowerEdge servers, using Intel Xeon Scalable processors.
While iDRACula is specific to Dell EMC PowerEdge products, and potentially some solutions built upon PowerEdge 13th generation and older servers, there is a broader discussion. Currently taking advantage of iDRACula requires a security policy lapse either by allowing physical access to a machine or remote access with a valid login credential. The broader discussion here is that Dell EMC is a leader in the industry for BMC security, so other server solutions are likely as if not more vulnerable to this type of activity. Vendors like Dell EMC, HPE, and even cloud players like Microsoft and Google are building silicon root of trust and signing apparatus to prevent firmware alternation like iDRACula going forward.
A Brief History of How We Got Here
Earlier this week, a STH forum member posted that they successfully replaced the iDRAC firmware with their own altered firmware. That post was taken down due to the sensitive nature of the topic, but the news was already out there.
Dell EMC learned of the STH forum post on Tuesday 25 September 2018 in the evening Pacific time, prior to its takedown. Although it was the evening, the company contacted the iDRACula poster and validated the methodology within 16 hours.
At that point, the company determined that it was a known vulnerability and a novel way to access the vulnerability. This is a family of vulnerabilities behind Dell EMC’s efforts to secure their platforms in new 14th generation PowerEdge servers using the new hardware features like a silicon root of trust. New PowerEdge servers may have hardware features to address this vulnerability, but there are millions of servers deployed today that will remain vulnerable without the new 14th generation PowerEdge hardware updates. As an editorial aside, it is a little hard to believe that state actors do not know of this type of vulnerability and have working methods for utilizing it.
Over the next few days, I stayed in contact with Dell EMC’s team and let them know that we would be running a story. I asked for a response after they had the opportunity to evaluate the severity and come up with a response plan.
I am not a security researcher. I will say that Dell EMC did a good job communicating with myself (knowing we would be running a story) and with the individual who took advantage of iDRACula. The server industry has seen a number of severe security issues this year, including the recent Intel L1TF / Foreshadow disclosure. Seeing Dell EMC’s security apparatus work quickly on this was welcome.
Over this week and several phone conversations, Dell EMC never asked me not to run this story, which I was slightly surprised about. If the company had asked, saying that this was too big of a security issue to go public, I would have obliged. Although this is undoubtedly a big story, we have a professional obligation in the industry to not chase page views on these sorts of stories if the company determines there is a clear and present threat to its users.
To get the editorial independence piece out of the way, as of this article, Dell EMC has never advertised with STH. Dell Technologies does cover some travel and press pass coverage for attending their shows such as Dell EMC World and VMworld for STH. Dell EMC also loans us servers for review such as the most recent Dell EMC PowerEdge R7415 Review. We do the most comprehensive server reviews. As of Q4, STH will review systems from virtually every server vendor on the Top 5 server vendors by revenue or units lists, including those who focus their distribution outside of the US. As an editorial policy, we do not allow vendors to see draft reviews or pieces like this before they go live. During the process, we asked Dell EMC for a comment for this article and they provided one.
Next, we are going to have an interview with one of the individuals that found iDRACula while trying to simply reduce the noise that their server made. We have Dell EMC’s response to the vulnerability. We will then discuss why this is a broader industry concern ending with some closing thoughts on the discussions you should be having within your IT team today to ensure you lower your risk profile.
Holy shit. Good job presenting. I’ve sent this to our team as a reminder we need to take this kind of security seriously and not go lax over time. I saw the post in the forums before it was taken down and I’d not put 2 and 2 together.
Dell is supposed to be the best at this………
I’ve know you could do this for awhile on regular ipmi servers.
Convenience comes with a price.
We were about to buy R730 but now we’re going to look at the R740 or R740xd.
So if I buy a new PowerEdge R630 today, ship it to our South American hub the person installing it tomorrow could do this to the server before I even see it is online? iDRAC is pre-boot so I’d have no way of knowing.
Is this an end of month Dell 14th gen sales promotion?
Was there a CVE submitted for this? Or does it not need one as an old firmware with a known vulnerability was used?
Aaron: a CVE is in the process of being submitted by dell, yes. While the remote process does involve rolling back to an older firmware, the issue/actual exploit lies in the fact the latest firmware has some holes that when used, allow you to then re-flash to the latest firmware and keep exploits in place undetected
Dell is still selling idrac 8 in new servers that can be exploited like this
Ok, so by handing a thief the key to your car and/or the key to your garage (because you chose to trust them)…. They can now take the motor out and modify it and put it back (hey that sounds fun)…. Helllllo you gave them the keys!!! They can turn it on or off anytime and have their way with it at will just by utilizing the key!! Or better yet, if I’m the owner and I hold the key I can have MY way with MY server, my choice of handing out the key or leaving the door open is the obvious security vulnerability at large. While a valid security flaw this is just a bit over fluffed. People need to learn more about enterprise best practices to prevent this, such as management vlans blocked from the internet, video surveillance and physical access control… rather than only buy 14th gen servers or learn the hard way.
John, you’re missing a huge part of the server deployment process, the supply chain. You can guard against physical access all you want once you have the equipment in-hand, but it doesn’t matter if anyone who had it previously had bad intentions, especially given as-of-now anything malicious they could potentially load will survive standard firmware re-flashes. Using your analogy, it’s more akin to buying a car that runs unsigned code on the engine control unit, and a pissed off employee at the dealer/tow company/etc loaded in their own modified firmware. The car seems perfectly fine, you keep it secure in the garage and nobody else has the keys, but one day that modified code is going to slam you into a wall.
Is this likely to happen to rinky-dink IT companies with not much to steal from? Doubtful, the chances are very low. is this likely to happen to large providers, financial institutions, etc? Yes, it already has happened. There is a reason Facebook, Microsoft, OpenBMC etc have spent millions solving the unsigned firmware issue, they have seen the damage it can do. 5 minutes of physical access to hardware to load something that will never be cleaned away is much easier for an entity to achieve in supply chains then you think, just ask any of the victims of previous exploits
This is why you never ignore defense in depth. Physically lock and secure your datacenters. IPMI traffic should be in an isolated vlan with no direct internet or end user access.
I was so excited to read about a new BMC vulnerability but this doesn’t apear to be anything new at all. Speaking strictly about the article, it seems to be a sensationalized piece about “scary BMCs oooh”. Nothing the security world doesn’t know about already. And please for the love of… stop naming these things. “integrated Dell Remote Access Controller unauthorized load access”… really??? You’re kinda grasping with that one.
If you’ve got physical access to a server you’ve already won. Not even talking about the iDRAC there are other interesting attack surfaces available as well (Internal SD card? Internal USB port? etc) but yeah the completely hidden and persistent ones are the best.
Speaking as someone who has actually discovered such a vulnerability in 2016 (CVE-2016-5685, local iDRAC8 root shell: https://www.cvedetails.com/cve/CVE-2016-5685/) I’m not sure how downgrading to an old, known to be vulnerable iDRAC release and exploting it is news. Did I miss something or is the -f[orce] flag that can be used to downgradeFW not a thing with the Dell Update Package anymore? I’m asking honestly. I haven’t checked. But that’d be an easy way to downgrade all the things by booting via Live USB.
I mean, if you managed to get root on the iDRAC on the old pre-patched version all by yourself (without hints) then in that case mad props to you for figuring it out. The fact that you’re now attacking a persistent file system and that this can’t be quickly fixed by reflashing to the same or newer version of the Firmware is known to Dell (and should be pretty obvious to anyone really) but clearly it wasn’t meant to be broadcast to the world…
Finally, the 14th Gen servers can and will be exploited… It will be harder to make these changes persistent but nothing says “challenge accepted” like a promise of complete protection from security bugs. If you have an iDRAC9 with a very early initial release of the FW you can drop to a root shell pretty easily too if you “invoke” some “debug” stuff: https://pbs.twimg.com/media/DIm-fhAUQAE8VaC.jpg
Luckily that was a very early FW release and was quickly patched…
Anyway, good on you for keeping BMCs on peoples radars as attack surfaces.
Is there an official dell support page for this? What is the CVE number?
THanks
“Please keep in mind you need either physical access to the PowerEdge server. To do this remotely, you must have existing iDRAC credentials.”
If someone has physical access to your servers you have bigger problems
Ditto iDRAC credentials..
@SA, apologies, just now seeing this. downgrading to a known exploitable firmware version is not the news, that’s however all STH was comfortable mentioning about the process and I don’t blame them. Now that a CVE has been assigned internally and a patch is coming out in a couple weeks I can probably say more: the larger exploit here is not being able to downgrade, but the process of downgrading, utilizing something that allows you to make arbitrary changes to the filesystem, and then upgrading back to the very latest “secure” firmware but keeping your changes in place undetected, persisting privilege elevation exploits regardless of how new firmware you flash to. According to dell any persistent changes like this should not have been possible, but clearly they are. There’s a couple other fun things too that wouldn’t have been good to publish months before a patch. I will probably publish more details on the github after the patch is released, but as you’ve guessed, since they don’t have hardware root of trust, one of these methods is unpatchable, and will always be possible.
And yes, very early versions of IDRAC9/14th gen firmware allowed you root access. It’s not of much use though, as you cannot load and persist anything. You can try, but on next power-on the entire boot chain from bootloader to entire EMMC flash is key checked against burned in keys in the proc and it will refuse to boot, as I’m sure you’re aware. That’s not to say there’s not some extremely niche bug in the signature checking chain somewhere, but it would certainly be far from arbitrary to exploit